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Abstract 

Background: Despite the achievements of previous research, caregiving assessments in severe mental illness 
should be crossculturally validated in order to define risk groups or to evaluate family work. This study reports on 
the psychometric properties of the European version of the Involvement Evaluation Questionnaire (IEQ-EU) in 
Portugal. 

Methods: A Portuguese translation of the IEQ-EU was developed according to the 'European Psychiatric Services: 
Inputs Linked to Outcome Domains and Needs' (EPSILON) group guidelines. We then studied 194 caregivers who 
were related to patients with schizophrenia spectrum disorders in psychiatric outpatient services. All relatives were 
assessed using the IEQ-EU. In order to describe the corresponding patients' sample, the majority (n = 162) was 
evaluated with the World Health Organization Disability Assessment Schedule (WHO-DAS II); 108 patients were also 
assessed with the Brief Psychiatric Rating Scale (BPRS) and the Global Assessment of Functioning (GAF). 

Results: The factor structure of the Portuguese version of the questionnaire was similar to the original; internal 
consistency was good, with Cronbach's a ranging from 0.71 to 0.87 in the IEQ-EU scales (total score and domains: 
tension, supervision, worrying, urging); test-retest reliability yielded intraclass correlation coefficients (ICCs) from 0.80 
to 0.94, concerning the same scores. Ecological validity was confirmed. Most caregiving consequences were 
reported on the worrying domain of the IEQ-EU. 

Conclusions: Validity and reliability of the Portuguese IEQ-EU translation were established. Specifically the four IEQ- 
EU subscale domains seem to be valid in Portugal. 

Keywords: Caregiver, crosscultural psychiatry, family member, measurement validation, questionnaires, 
schizophrenia 



Introduction 

Despite huge amounts of high-quality research related 
to caregiving in severe mental illness (see for example 
[1-4]), a need exists for the crosscultural validation of 
caregiver instruments, and for the development of 
national norms regarding assessments. Without better 
knowledge of the intercultural validity of measures and 
local standards, international comparisons will be under- 
mined by doubts regarding the origin of differences 
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between scorings (for example, sampling vs real cultural 
differences) [5]. 

The European Psychiatric Services: Inputs Linked to 
Outcome Domains and Needs (EPSILON) study was a 
crossnational, cross-sectional survey [6,7], which com- 
pared characteristics, needs and quality of life of people 
with schizophrenia and their caregivers in five European 
countries (Denmark, England, Italy, The Netherlands 
and Spain). Standardized versions of related key research 
instruments were produced, including the Involvement 
Evaluation Questionnaire (IEQ) for the assessment of 
caregiving consequences [8]. 
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The IEQ measures the consequences of psychiatric dis- 
orders for relatives, friends or other significant persons 
involved, being a well established tool for assessments of 
caregiving impact in psychotic, mood or mixed disorders 
[5,9]. In the EPSILON study the English, Danish, Italian 
and Spanish translations were validated [8]. Over the last 
decade ten more translations were made. Validation data 
on the German, Swedish, Malawi, Chinese, Polish and 
Arabic versions are available [10-15]. The Finnish, French 
and Greek versions have not been validated yet [16]. A 
previous version of the IEQ was used in Portugal [17], but 
to date the IEQ-EU has not been adequately validated in 
Portuguese populations. 

Therefore, our aim was to validate and test the reliability 
of a Portuguese translation of the IEQ-EU, according to 
the EPSILON study methodology, in Portuguese caregivers 
of psychotic outpatients. In this paper, we document the 
validity of the IEQ-EU scales in Portugal (focusing on eco- 
logical, face value and content aspects), along with their 
test-retest reliability and internal consistency. 

Methods 

Study design and participants 

After the development of the IEQ-EU Portuguese trans- 
lation and pretests, factor structure and internal consis- 
tency were determined. 

We studied a non-randomized sample of primary care- 
givers of outpatients with chronic psychosis, who were in 
contact with mental health services in Lisbon (n = 194). 
These services were Hospital S. Francisco Xavier, a public 
hospital, and Clinica Psiquiatrica de S Jose, a non-profit 
facility. The majority of participants (n = 108) formed the 
baseline sample of the 'FAmilies of people with PSychotic 
disorders' (FAPS) survey, for which preliminary results 
have been presented [18]. In this paper, only the IEQ-EU 
data will be used. Results for the FAPS baseline study are 
reported in detail elsewhere [19] (Goncalves-Pereira M, 
Xavier M, van Wijngaarden B, Papoila AL, Schene AH, 
Caldas de Almeida JM: Impact of psychosis in a Portu- 
guese population: a cross-cultural exploration of burden, 
distress, positive aspects and clinical-functional corre- 
lates, unpublished). Test-retest reliability was evaluated 
in those caregivers who agreed to collaborate (n = 50), by 
asking them to complete the IEQ-EU a second time 
within a 2 to 3 week period. The overall sample size was 
not defined a priori, yet it should allow for the study of 
the IEQ-EU psychometrics, including factorial validity. 
For that purpose, the FAPS sample was enlarged with 
additional participants. 

In the FAPS survey, 108 patients with International 
Classification of Diseases, 10th edition (ICD-10) Diagnos- 
tic Criteria for Research (DCR) schizophrenia (F20), schi- 
zoaffective (F25) or delusional disorder (F22), were 
included. Regarding all non-FAPS patients in this study 



(n = 86), a simple ICD-10 clinical diagnosis of one the 
above-mentioned disorders was the condition to be 
included. Cases with coexisting learning disability, 
organic disorders, or inpatient treatment episodes in the 
previous 2 months were excluded. 

Primary caregivers were then approached. All of them 
were family to the patient, but relatives will be referred 
to as 'caregivers' for the remainder of this article. The 
protocol was approved by local ethical committees. 
Informed consent was obtained from all participants. 

The IEQ-EU and its Portuguese translation 

The IEQ-EU is an 81-item questionnaire to be completed 
by any caregiver who, during the past 4 weeks, had at 
least 1 h a week contact with the patient. The question- 
naire consists of seven modules: (1) demographics of 
patient and family, and general clinical data concerning 
the patient; (2) caregiving consequences of psychiatric 
disorders; (3) extra financial expenses; (4) the General 
Health Questionnaire, 12-item version (GHQ-12), used 
as a general measure of distress [20]; (5) professional 
help for patient's relative; (6) consequences for patient's 
children; (7) open question for additional remarks. Mod- 
ule 2 is the core module (31 items), where items refer to 
all kinds of encouragement and care the caregiver has to 
provide to the patient, to supervision of the patient's dan- 
gerous behaviors, to interpersonal problems between 
patient and caregiver, and to the caregiver's worrying (for 
example, 'how often have you worried about your rela- 
tive's future?'). These items are scored on a five-point 
Likert scale (0 = never, 1 = sometimes, 2 = regularly, 3 = 
often, 4 = (almost) always), and the time frame is the 
prior 4 weeks. The caregiver's coping and subjective bur- 
den are also assessed in individual items. In research use, 
27 of the 31 core module item scores can be summarized 
in a total or sum score, and 4 subscales or domains: ten- 
sion (9 items), supervision (6 items), worrying (6 items), 
and urging (8 items). Two items (no. 29 and no. 43) load 
on two subscales [5,21]. 'Tension' refers to the strained 
interpersonal atmosphere between patient and relatives; 
'supervision' to the caregiving tasks of ensuring and 
guarding related to, for example, the patient's intake of 
medicine or dangerous behaviors; 'worrying' to painful 
cognitions and concerns about patient's safety or future; 
and 'urging' to issues related to activating and motivating 
the patient. An overview of the core items of the IEQ-EU 
is given in the Results section. 

The IEQ Portuguese translation was developed in 
accordance with international conventions and EPSILON 
guidelines, in order to maintain face and content validity 
[22]. All modules had to be translated except the GHQ- 
12, for which a Portuguese translation was available from 
the publisher [23]. A first draft concerned the translation 
of the English IEQ-EU by one of the authors (MGP), who 
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also was involved in the Portuguese version of the origi- 
nal IEQ [17]. This translation was refined by a Dutch 
native Portuguese-fluent lay contributor using the Dutch 
IEQ-EU. A focus group involving caregivers, and a dis- 
cussion group involving mental health professionals and 
researchers, were both conducted in order to check the 
quality and acceptability of the translation. Drafts were 
subsequently refined and a back translation was per- 
formed by a Dutch native Portuguese-fluent professional 
translator. When revised by one of the authors of the ori- 
ginal tool (BvW), only minor corrections were made. 

We chose not to report the GHQ-12 results here, as 
only FAPS participants completed this questionnaire. 
Moreover, these assessments are relevant to construct 
validation but not to define the psychometric properties 
related to the core IEQ-EU. 

Other measures 

Patient clinical data not covered by the IEQ-EU, such as 
number of previous admissions, were collected in 
interviews. 

Regarding the FAPS baseline sample, patients' symp- 
toms were assessed with the extended (24 item) Brief 
Psychiatric Rating Scale (BPRS) [24]. BPRS items are 
coded into seven categories (1 = no symptoms, to 7 = 
extremely severe). Patients' disability was assessed 
through interviewing caregivers and collecting other 
sources of information with the World Health Organiza- 
tion Disability Assessment Schedule II (DAS-II) [25], 
producing global evaluations from 0 ('excellent or very 
good adjustment') to 5 ('severe maladjustment'). Finally, 
the Global Assessment of Functioning (GAF) scale [26] 
was applied, in a continuum from 0 to 100. The DAS 
interview was also used for 54 patients in the non-FAPS 
sample, so that data on the global evaluation section 
were available for 162 cases. These clinical and functional 
assessments were made by trained research assistants, all 
of them mental health professionals. 

Statistical analysis 

Descriptive statistics and non-parametric tests were used 
as required. Patient and caregiver characteristics are pre- 
sented as frequencies and percentages for categorical 
data, and as mean or median, standard deviation (SD), 
range and minimum/maximum values for continuous 
variables. The 95% confidence intervals (CI) for the mean 
value were calculated whenever appropriate. 

Principal component analysis was used to conduct an 
exploratory factor analysis on the IEQ scores. To verify 
the appropriateness of the factor analysis, three techniques 
were used for the assessment of the psychometric ade- 
quacy of the correlation matrix: (1) Bartlett's test of spheri- 
city, evaluating the hypothesis that the correlation matrix 
is an identity matrix (that is, there is no correlation among 



the items); rejection of this hypothesis suggests that data 
are appropriate for factor analysis; (2) inspection of the 
off-diagonal elements of the anti-image correlation matrix 
(that contains the negatives of the partial correlation coef- 
ficients), enabling us to quantify individual measures of 
sampling adequacy (MSA) and to conclude that the corre- 
lation matrix is factorable whenever the absolute values of 
those elements are small; (3) Kaiser-Meyer-Olkin (KMO) 
statistic, an overall MSA that varies between 0 and 1 (high 
values for this statistic indicate that the data are unsuitable 
for factor analysis). Four factors were chosen a priori in 
order to provide comparable results with similar studies. 
The Varimax rotation technique with Kaiser normalization 
was used. It was postulated that factor loadings should be 
> 0.40. In case of several loadings > 0.40 in a particular 
item, this item would be used for more than one factor 
score. 

For IEQ scores, test-retest reliability was assessed cal- 
culating intraclass correlation coefficients (ICCs) and 
95% CI; Cronbach's a values were computed for internal 
consistency. Regarding correlation studies, Spearman's 
coefficients were used when a linear association was 
present between continuous variables. The significance 
level of a = 5% was considered. All data were entered 
and analyzed using SPSS for Windows V.15.0 (SPSS 
Inc., Chicago, IL, USA). 

Results 

Demographics and clinical data regarding the patients 
and their primary caregivers are first described. The 
mean age of the patients in the total sample (n = 194) 
was 35.6 (SD 9.5) years. Most were male (63.4%), had a 
diagnosis of schizophrenia (70.1%) or other chronic psy- 
chosis (29.9%), a number of psychiatric admissions ran- 
ging from none to 16 (median 2.0), and a duration of 
illness ranging from 1 to 63 years (median 10.0). DAS 
mean (SD) scores were 2.6 (0.9) (n = 162). In the FAPS 
baseline sample (n = 108), BPRS mean (SD) scores were: 
1.8 (0.5); and GAF mean (SD) scores were: 52.6 (13.8). 
These FAPS patients had ICD-10 DCR schizophrenia in 
the majority of cases (88.9%), mainly of the paranoid 
type, F20.0 (76.9%). 

The corresponding caregivers' characteristics (n = 194) 
are shown in Table 1. Most were female (67.5%), parent 
of the patient (71.1%), and living with him/her (89.7%). 
All completed the IEQ-EU. 

General data on the psychometrics of the IEQ Portuguese 
version 

Factor analysis was conducted on this sample. Accord- 
ing to Bartlett's test (value = 1,811.66 and P < 0.001) the 
correlation matrix was not an identity matrix and was 
therefore suitable for analysis. The KMO statistic was > 
0.70 (0.798), the MSA values for all the individual items 
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Table 1 Caregivers' demographics and caregiving arrangements 



Caregiver demographic 



Value 



Mean (SD) age, years 
Gender 

Mean (SD) monthly net income 
Marital status 
Living situation 

Relationship to patient 

Lives with patient? 

Relationship characteristics: 

Mean weekly contact > 32 h 

Mean (SD) hours of personal contact 

Mean (SD) no. days lived together in 
previous month; median 



57.7 (13.7); range: 18 to 86 

Male: 63 (32.5%); female: 131 (67.5%) 

3.8 (1.5) 

Married: 123 (63.4%); single: 15 (7.7%); divorced: 31 (16.0%); widowed: 25 (12.9%) 

Extended family: 14 (7.2%); with spouse/children: 156 (80.4%); with parents/brothers: 14 (7.2%); with 
other relatives: 4 (2.1%); with others: 3 (1.5%); alone: 3 (1.5%) 

Mother/father: 138 (71.1%); sibling: 24 (12.4%); partner: 20 (10.3%); other relative: 11 (5.7%) 
Yes: 1 74 (89.7%) 

1 59 (82.0%) 
5.6 (1.1) 

24.1 (10.1); 28.0 



n = 194. Monthly net income is coded into six range scores in item 14 of the IEQ (1, minimum, to 6, maximum); hours of personal contact are coded into six 
range scores (1 = less than 1 h a week to 6 = more than 32 h a week). 



were > 0.60 (89% were > 0.70) and the absolute values 
of the off diagonal values were very low. Therefore, the 
analysis seemed appropriate. The factor analysis resulted 
in a very similar solution to the original one [5]. This 
solution is presented in Table 2 and it accounted for 
47% of the total variance. The value of lost expected 
variance for IEQ-EU totals and subscales was acceptable 
(13%) if we complied with the original subscales, so we 
decided to use these in the remainder of the analysis. 

The distribution of IEQ-EU scales sum item scores 
and their internal consistency are presented in Table 3. 
Mean scores (SD), for each IEQ domain and total score, 
were: tension 0.88 (0.57); supervision 0.45 (0.58); worry- 
ing 2.3 (0.73); urging 1.07 (0.75); IEQ total score 1.12 
(0.54). Overall, Cronbach a values were 'substantial' [7], 
being higher for the IEQ sum score (0.87) than for IEQ- 
EU subscales (0.71 to 0.74). 

Tension and worrying, the two interpersonal domains, 
had a substantial correlation (r s = 0.71; P < 0.001), and 
there was a moderate coefficient between tension and 
urging (r s = 0.53; P < 0.001). All other correlations 
between domains were lower than 0.49, favoring the 
adopted original factor structure. 

Test-retest reliability was assessed for 50 participants. 
The ICC for the IEQ-EU total score was 0.93 (95% CI 
0.88 to 0.96, P < 0.001), while for subscales ICCs were: 
0.88 (95% CI 0.79 to 0.93, P < 0.001) for tension, 0.79 
(95% CI 0.63 to 0.88, P < 0.001) for supervision, 0.94 
(95% CI 0.89 to 0.97, P < 0.001) for worrying, and 0.89 
(95% CI 0.80 to 0.94, P < 0.001) for urging. 

Item-level results 

Three IEQ-EU core module items which are not used to 
compute domain or total scores provided specific results 
on being able to pursue own activities and interests, get- 
ting used to mental illness, and self-perception of lack of 



coping ability. These three items were also scored from 0 
(lower level) to 4 (higher level). Results (mean (SD)) were 
as follows: 1.5 (1.5) for pursuing own activities (median 1, 
meaning 'sometimes'); 1.9 (1.3) for getting used to pro- 
blems (median 2, meaning 'fairly well'); and coping ability 
1.9 (1.2) (median 2, meaning 'regularly felt able to cope'). 

Ecological validity and acceptability of the IEQ-EU 

The ecological validity of the IEQ Portuguese version is 
sufficient. Considering the FAPS baseline sample, the 
overall response rate was 100%, no questionnaires had to 
be discarded from analysis, and the applicability was also 
sufficient. However, caregivers varied in their ability to 
complete the IEQ-EU on their own, reflecting heteroge- 
neity of literacy levels. In total 61 respondents (56.5%) 
were able to fully self-complete questionnaires, 32 
(29.6%) required significant help, and 15 (13.9%) had to 
be interviewed due to limited literacy. According to inter- 
viewers' impressions, assessments were easily conducted 
and item formulation and understandability seemed ade- 
quate, even for illiterate respondents. All but five care- 
givers fully completed the IEQ. 

Discussion 

This study replicates a part of the EPSILON study meth- 
odology for adequate testing of validity and reliability of 
the IEQ-EU in a South European setting (Portugal), with 
a considerable sample size. The IEQ had been originally 
developed in the north of Europe (The Netherlands). 

Regarding patients' characteristics, they are typical of a 
clinical sample of chronic psychotic users in mental 
health outpatient public practices in Portugal [1,17]. The 
levels of psychopathological symptoms and global func- 
tioning of these users are similar to the ones described in 
previous studies on caregiving [5]. Mean disability levels 
are around the 'poor adjustment' DAS category, again in 
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Table 2 Principal component analysis on Involvement Evaluation Questionnaire (IEQ) items 



Items 



Factors 



Tension 



Supervision 



Worrying 



Urging 



Percentage of variance (total = 46.55%) 24.95% 6.07% 

How often during the past 4 weeks: 

has the atmosphere been strained between both 0.786 
been annoyed by relative's behavior 0.736 
thought of moving out 0.7 
has your relative caused a quarrel 0.679 
felt threatened by your relative 0.564 0.439 

carried out tasks normally done by your relative 0.49 
heard from others they have been annoyed 0.478 
have you encouraged your relative to take proper care 
to eat enough 

ensured that your relative has taken required medicines 
encouraged your relative to undertake some kind of activity 

helped your relative to take proper care of him/herself 0.49 
ensured your relative received sufficient sleep 
worried about the help/treatment your relative receives 
worried about your relative's safety 
have you worried about your relative managing financially 
worried about your relative's future 
worried about your own future 
worried about your relative's general health 

have your relative's mental health problems been a burden 0.427 
have you guarded your relative from taking illegal drugs 0.713 
guarded your relative from drinking too much alcohol 0.627 
guarded your relative from self-inflicted harm 0.621 
guarded your relative from committing dangerous acts 0.529 



0.47 

0.774 

0.727 

0.684 

0.46 

0.436 



7.42% 



0.7 

0.64 

0.615 

0.61 

0.577 

0.554 

0.443 

0.406 



Bold type indicates the factor to which the item was allocated in cases where it loaded > 0.4 in two or more factors; items loading < 0.4 in any factor are not 
shown (20: accompanying on outside activities; 28: encouraging to get up in the morning; 29: sleep disturbance due to relative's behavior). 



agreement with our assumption that the sample is prob- 
ably representative of populations with chronic psychosis 
in Portuguese services [1]. Moreover, the corresponding 
caregivers' characteristics and caregiving arrangements 
are close to the usual pattern in South Mediterranean 
countries [1,5]. 

Overall, validity of the Portuguese IEQ-EU translation 
has been established. In this study, the ecological, face 
and content validity of the questionnaire were grounded 
on quality assurance of the translation, and on replication 



of the EPSILON methodology. Criterion and construct 
validity, and sensibility to change, were not directly 
approached here. However, the FAPS survey contributed 
to their specific testing in Portugal by comprehensively 
assessing 108 of these 194 patient-caregiver dyads, as 
reported elsewhere [18,19]. Construct validity is particu- 
larly sound [19] (Goncalves-Pereira M, Xavier M, van 
Wijngaarden B, Papoila AL, Schene AH, Caldas de 
Almeida JM: Impact of psychosis in a Portuguese popula- 
tion: a cross-cultural exploration of burden, distress, 



Table 3 Involvement Evaluation Questionnaire (IEQ) totals and domains: distribution of scores and internal 
consistency 





Sum item scores (0 to 4) 






Cronbach's a 




Minimum to maximum 


Mean (95% CI) SEM 


SD 


Coefficient 


95% CI (unilateral) 


Tension 


0 to 31 (0 to 36) 


7.92 (7.19 to 8.64) 0.37 


5.11 


0.74 


> 0.69 


Supervision 


0 to 21 (0 to 24) 


2.70 (2.21 to 3.18) 0.25 


3.45 


0.71 


> 0.65 


Worrying 


0 to 24 (0 to 24) 


13.79 (13.04 to 14.54) 0.381 


5.31 


0.73 


> 0.68 


Urging 


0 to 27 (0 to 32) 


8.56 (7.71 to 9.41) 0.43 


6.01 


0.74 


> 0.69 


IEQ sum score 


0 to 84 (0 to 1 08) 


30.23 (28.16 to 32.31) 1.05 


14.65 


0.87 


> 0.85 



n = 194; minimum and maximum possible scores are given in brackets. 
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positive aspects and clinical-functional correlates, 
unpublished). 

Our data on psychometric properties ensure consider- 
able reliability. Concerning internal consistency, alphas 
were slightly lower than in the EPSILON global sample, 
but some of them were more satisfactory than in some of 
the EPSILON centers [8] . Regardless of comparisons, they 
were 'substantial' according to the EPSILON convention 
[7] and 'good', according to Streiner and Norman, for this 
kind of scales [27]. ICCs on test-retest were invariably 
'substantial' to 'high' [7]. We emphasize that the 50 
respondents constituted a larger subsample for this kind 
of testing than most in the EPSILON study [8]. 

Our results also suggest that the original IEQ-EU factor 
structure can be preserved, as its use in our sample did 
not lead to too much loss of explained variance. Rank 
order of the four IEQ-EU domains was the same as in the 
EPSILON and previous Dutch studies: the mean score of 
worrying is the highest, followed by urging, interpersonal 
tension and supervision [5,21]. We consider this finding as 
another sign of validity. 

Therefore, in Portugal, the IEQ-EU seems to cover the 
same caregiving domains that have been described for 
other countries, and instrument bias in the assessment of 
differences in caregiving consequences appears unlikely. In 
the FAPS baseline assessments, ecological validity of the 
IEQ-EU was evident, with high response rates and very 
few missing data. 

Limitations of the study 

We did not use randomization procedures so sampling 
bias cannot be fully discarded. Therefore, despite the 
strong impression that our participants represent a typical 
sample of caregivers of chronic psychotic patients in Por- 
tugal, one must theoretically recognize limitations to the 
generalizability of our findings. We also did not use a semi 
structured psychiatric interview, although a robust clinical 
diagnosis ascertainment was ensured for the majority of 
cases. 

All assessments were conducted in clinical settings, 
and postal means were not used. This may have posi- 
tively influenced response rates and questionnaires' 
acceptability. 

Conclusions 

A heavy negative impact of caregiving in severe mental 
illness has been acknowledged once more in a regional 
sample [28]. There is a need to continuously address 
vulnerable caregivers, and feasible risk assessment rou- 
tines must be pursued. 

The Portuguese version of the IEQ-EU is valid and 
reliable for research use, but its clinical usefulness 
remains a challenging topic. Regardless of this, the 



available evidence sustains the maintenance of the origi- 
nal IEQ factor structure in Portugal. 
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